feat: Add support for List data types #39
Conversation
mgrazianoc
force-pushed
the
mgraziano/list-support
branch
from
28f225d to
0253f70
Compare
There was a problem hiding this comment.
Pull Request Overview
This PR extends the Arrow integration to support List data types while refactoring nested type handling. Key changes include renaming ArrowNestedType to ArrowTypeStruct, adding a new ArrowTypeList and corresponding builders/arrays, and updating relevant tests to cover both primitive and nested lists.
Reviewed Changes
Copilot reviewed 11 out of 11 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| Arrow/Tests/ArrowTests/TableTests.swift | Updated closure signature and return type for struct builders |
| Arrow/Tests/ArrowTests/IPCTests.swift | Updated schema construction to use the new ArrowTypeStruct |
| Arrow/Tests/ArrowTests/ArrayTests.swift | Added tests for ListArray functionality with both primitive and nested lists |
| Arrow/Sources/Arrow/ProtoUtil.swift | Refactored type creation for struct and added support for list fields |
| Arrow/Sources/Arrow/ArrowWriter.swift | Updated list handling by replacing ArrowNestedType with ArrowTypeStruct |
| Arrow/Sources/Arrow/ArrowType.swift | Renamed ArrowNestedType → ArrowTypeStruct and added ArrowTypeList |
| Arrow/Sources/Arrow/ArrowReaderHelper.swift & ArrowReader.swift | Added list data loading functions |
| Arrow/Sources/Arrow/ArrowBufferBuilder.swift | Replaced nested type usage and added ListBufferBuilder |
| Arrow/Sources/Arrow/ArrowArrayBuilder.swift | Introduced ListArrayBuilder and updated builder resolution |
| Arrow/Sources/Arrow/ArrowArray.swift | Added ListArray implementation with subscript and asString |
kou
changed the title
|
@abandy You may want to review this. |
Will do, thanks! |
mgrazianoc
force-pushed
the
mgraziano/list-support
branch
2 times, most recently
from
0f11175 to
c4f7ef0
Compare
Co-authored-by: Sutou Kouhei <[email protected]>
Co-authored-by: Sutou Kouhei <[email protected]>
Co-authored-by: Sutou Kouhei <[email protected]>
Co-authored-by: Copilot <[email protected]>
@mgrazianoc Initial review it lgtm! I will try to find sometime later this week to go over again. |
|
Hi @abandy , do you think we could have this merged by the next weeks? Anything I can help, please let me know. |
Rationale within the changes
This PR refactors and extends support for nested types in the Arrow integration. The current implementation of
ArrowNestedTypeis tailored primarily for data structs, as seen inStructBufferBuilder. However, it lacks broader support and certain expected functionalities, such asloadStructArrayBuilder.To address this, the following improvements have been made:
ArrowNestedTypetoArrowTypeStructto align with naming conventions used elsewhere in the codebase.ArrowTypeList, including nested lists.For simplicity, instead of introducing a dedicated subtype for lists, this PR uses an interface of
[Any?]?. If this approach proves insufficient, there are more explicit alternatives that can be explored.NOTE: Work on
ArrowCExporterandArrowCImporterhas been intentionally deferred. These components require a deeper understanding of memory ownership and child parsing, and I believe it's better to be addressed in a future PR, unless it's strict necessary.What's Changed
ArrowNestedType -> ArrowTypeStruct.ArrowTypeList, including nested lists.ListArraywith basic.asStringformatting.ListArrayBuilder.ArrowArrayBuilderto support the.listtype.loadStructArrayBuilderandloadListArrayBuilder.ListBufferBuilder.ArrowReader.loadListData.makeListHolder.Are these changes tested?
Tests are included in
ArrayTests.swift. It's also working on internal applications, including integration withArrowFlight.Closes #16.